What symptoms would indicate a 'split-brain' style problem in a distributed Qdrant cluster, and how does the Raft-based consensus layer prevent it?

Split-brain in a distributed system means two parts of the cluster each believe they are the authoritative source and accept conflicting operations. In a Qdrant cluster, the symptoms would be: two nodes both claiming to be the Raft leader, conflicting metadata views (different nodes reporting different collections, shard layouts, or configs), writes accepted on one side that are not visible on the other, and inconsistent results for the same query depending on which node it hits. These symptoms are alarming because they indicate that the cluster's metadata is diverging, and once metadata diverges, the data plane cannot be reconciled automatically. In practice, well-implemented Raft prevents split-brain by design, so the symptoms above should not occur in a correctly functioning cluster. If they do occur, the most likely cause is a bug, a misconfiguration, or a network partition that the Raft implementation did not handle as expected - not a fundamental limitation of the protocol.

The mechanism that prevents split-brain is Raft's quorum requirement. To become leader, a node must receive votes from a majority of the cluster's voting members. To commit a log entry, the leader must replicate it to a majority of members. Two different nodes cannot both become leader in the same term because that would require two different majorities, and in a cluster of N nodes, two majorities always overlap. Two different leaders in different terms cannot both commit entries because a new leader is elected only after a majority has acknowledged the previous leader's term has ended. The result is that at most one leader can commit entries at any time, and any committed entry is durable across a majority of nodes. When a partition occurs, the side with the majority continues to operate and can elect a leader; the side without the majority cannot commit new entries and effectively becomes read-only or unavailable for metadata operations. When the partition heals, the minority side discovers the higher term and catches up as a follower. This is the standard Raft safety property and it is what prevents split-brain.

The trade-off is that the quorum requirement means a cluster cannot make metadata changes when it does not have a majority of nodes available. In a 3-node cluster, if 2 nodes are down, the remaining node cannot elect a leader and cannot commit new metadata operations. Existing shards may continue to serve reads and writes at the data plane, but no new collections can be created, no configs can be changed, and no rebalancing can happen. This is the price of split-brain prevention, and it is why the recommended minimum cluster size is 3 nodes for production. The common mistake is to assume that a Qdrant cluster can continue full operations with a minority of nodes. It cannot, for metadata operations. The second mistake is to confuse data-plane availability with metadata-plane availability. A cluster can be serving queries perfectly while being unable to create a new collection because it lacks metadata quorum. The third mistake is to interpret a metadata operation failure as a split-brain event when it is actually a quorum loss. Split-brain is prevented; quorum loss is an expected behavior. Version note: the exact Raft implementation and the behavior during partitions have changed across Qdrant releases. Some versions have more sophisticated handling of partial failures; others are simpler. If you rely on metadata availability for your SLA, test the behavior with a forced partition on your version.

Version-dependent: Qdrant's consensus implementation has evolved. Early versions used a single Raft group for the whole cluster; more recent versions have moved toward per-collection consensus and more sophisticated topology management. The behavior during partitions, the exact error messages for quorum loss, and the recovery procedure have changed. If you are designing for high availability, test the specific failure scenarios (partition, leader loss, quorum loss) on your version rather than relying on a general description of Raft.